What

Replaces the fixed wall-clock catch-up budget in integrationTests/stress/largeCatchup.test.mjs with a throughput floor, and adds host-pressure counters to the run log, the job summary, and the uploaded log artifact.

Why — verdict on host contention: refuted, with runner-side evidence

The nightly Large-Data Stress Tests / large-catchup job failed on 2026-07-25 and 2026-07-26 at the identical harper-pro head SHA (86f2955e) and identical pinned core pointer (cda8d63f) that passed on 2026-07-24. Both failures were a wall-clock miss of the 1800s budget at ~87–90% converged, not a wedge.

The working hypothesis going in was host-level I/O and CPU contention on the shared harper-bench box. The runner-side evidence does not support it. sar history on that host (/var/log/sysstat/sa24, sa25, sa26), aligned to each job's startedAt/completedAt:

Every contention indicator is at or below the passing night's on both failing nights. 07-26 in particular: the box had been rebooted the previous evening, was running ~700 fewer processes with half the committed memory, and the disk was at half the passing night's utilisation — and the run still failed.

Two further discriminators point the same way:

1. A's write phase held up. 07-24 pass 269.3 MB/s, 07-25 fail 251.5 MB/s, 07-26 fail 194.4 MB/s. A host-wide slowdown would hit the write path too; 07-25 was 7% off the passing night on writes while catch-up was 2.3× slower.
2. The fleet was active on all three nights. The dispatch worker ran codex advise/review jobs during the passing window as well as both failing windows, so its presence does not separate pass from fail.

What is actually happening

Catch-up throughput is bimodal on identical code. Reconstructing the per-poll progress from the job logs:

• 07-24 (pass): plods at ~1440 records per 10–20s for the first 648s (100 → 40 200), then accelerates sharply — 40 200 → 102 620 in 223s, ~4.5× the earlier rate. Total 900.3s, 11.4 MB/s.
• 07-25 / 07-26 (fail): never enter the fast mode. Steady ~1440-record bursts for the whole 1800s, ending at 91 280 and 94 180 records — 4.9 and 5.1 MB/s. Extrapolating their own observed rate, both would have converged around 2000–2100s.

Measured 10 GB distribution: 13.5, 11.4, 10.0 MB/s (converged) and ~5.1, ~5.2 MB/s (slow mode). The old 1800s budget sits at 5.7 MB/s equivalent — right in the middle of that distribution, which is why the job coin-flips.

I have not root-caused the fast/slow mode split; that is filed as a follow-up (see Findings in the dispatch). This PR makes the signal trustworthy, it does not fix the throughput variance.

The new criterion

-	// Budget: longer offline = larger backlog = more catch-up time needed.
+	const CATCHUP_FLOOR_MBPS = Number(process.env.HARPER_STRESS_LARGE_CATCHUP_FLOOR_MBPS ?? 2.5);
+	const CATCHUP_BASELINE_MBPS = Number(process.env.HARPER_STRESS_LARGE_CATCHUP_BASELINE_MBPS ?? 10);
+	const CATCHUP_GRACE_SECS = Number(process.env.HARPER_STRESS_LARGE_CATCHUP_GRACE_SECS ?? 120);
 	const CATCHUP_BUDGET_SECS = Number(
-		process.env.HARPER_STRESS_LARGE_CATCHUP_BUDGET_SECS ?? Math.max(600, TARGET_GB * 180)
+		process.env.HARPER_STRESS_LARGE_CATCHUP_BUDGET_SECS ??
+			Math.max(600, Math.ceil((TARGET_GB * 1024) / CATCHUP_FLOOR_MBPS) + CATCHUP_GRACE_SECS)
 	);

The deadline is no longer a chosen constant — it is the time the floor rate needs for this dataset, plus a grace for B's restart/reconnect. Missing it therefore means, and now reads as, "average catch-up throughput was below the floor". At 10 GB that is 4216s (~70 min), inside the job's existing timeout-minutes: 180; a healthy run still exits as soon as it converges (900–2100s observed), so typical nightly wall-clock is unchanged.

Floor = 2.5 MB/s, chosen as half the slowest healthy run measured (5.1 MB/s), not a round number.

What would still catch a genuine 2× replication regression: a 2× slowdown from the slow mode (5.1 → 2.55 MB/s) trips the floor and hard-fails. A 2× slowdown from the fast mode (11.4 → 5.7 MB/s) does not — and honestly cannot, because that is indistinguishable from the slow mode this codebase already produces. That case is covered by the tracked metric instead: any run under CATCHUP_BASELINE_MBPS (10 MB/s) emits a ::warning annotation on the run page and records the number in the metrics artifact, so a sustained downward trend is visible before it ever reaches the floor. Closing that gap properly requires eliminating the bimodality, not a tighter threshold.

Unchanged hard failures: the STALL_SECS (300s) wedge / no-progress guard and its wording from 806a829, over-replication (lastCount > targetCount), OOM and uncaughtException log scans, the RSS_CAP_MB and ANON_CAP_MB memory ceilings, and the A-process-exit watcher during the write phase.

Host-side observability

stressShared.mjs gains readHostCounters() / hostCounterDelta() / sampleHostCounters() / summariseHostSamples() / formatHostCounters(). /proc/stat, /proc/diskstats, /proc/loadavg, /proc/pressure/* and /proc/meminfo are not namespaced by Docker, so the job container reads the whole box through them — exactly the signal the existing in-cgroup stats cannot see (they were identical on pass and fail nights, which is why attribution was impossible from the artifacts).

Emitted as: a suffix on every catch-up poll line; a write host: line and a host (catch-up): + host peaks: pair; inside the assertion failure message itself; and as large-catchup-metrics.json in HARPER_INTEGRATION_TEST_LOG_DIR (i.e. the large-stress-logs-large-catchup-node-24 artifact) plus a GITHUB_STEP_SUMMARY table.

[large-catchup] host (catch-up): load=2.63 iowait=2% steal=0% psi(cpu/io/mem)=0%/2%/0% hostIO=0r/516w MB/s memAvail=24446 MB
[large-catchup] host peaks: load1=2.63 iowait=2% psiIo=2% psiCpu=1% minMemAvail=24435 MB

Test notes

Verified locally at reduced scale per the test's docstring (HARPER_RUN_STRESS_TESTS=1 HARPER_STRESS_LARGE_DATA_GB=1), on the harper-bench host itself:

• Pass path — converges in 6.0s at 171.7 MB/s, host counters emitted at every stage, large-catchup-metrics.json and the job-summary table written. pass 1 / fail 0.
• Floor-breach path — forced with HARPER_STRESS_LARGE_CATCHUP_BUDGET_SECS=2; fails with B averaged 0.0 MB/s, under the 1024.0 MB/s catch-up floor (0/10486 records in 2s) — slow replay, not a wedge; a wedge would have tripped the 300s no-progress guard. Host during catch-up: load=1.07 iowait=2% ….
• Below-baseline warning path — forced with HARPER_STRESS_LARGE_CATCHUP_BASELINE_MBPS=1000; emits the ::warning annotation and still passes.

The floor quoted in messages is ENFORCED_FLOOR_MBPS, derived back out of the actual budget, so it stays truthful when the 600s minimum or an explicit budget override dominates (hence 1024.0 MB/s in the forced case above — that is the arithmetic consequence of a 2s budget, not a config value).

A full 10 GB run was not attempted locally; the wedge guard, over-replication guard and memory caps are untouched code paths.

Not touched

Per the brief: the core submodule pointer is unchanged, Sync Core PR #610 was neither merged nor approved, and largeClone plus the other stress matrix entries are untouched. stressShared.mjs changes are additive — existing exports are unmodified, so the other stress tests are unaffected.

Related context (not addressed here — Kris's merge call)

harper-pro main's core pointer (cda8d63f, 2026-07-22) is 23 commits behind harper main and is missing both replication resume-cursor fixes — 9aca3264c "fix(replication): stop the resume-cursor write freezing the apply worker" and e47f9ca68 "fix(replication): abort the cursor transaction when its commit fails" — which closed #603, whose signature was the apply worker freezing up to 101s under RocksDB write stall. That is the same family as the 72–90s inter-batch stalls still visible in these runs, so the nightly is currently exercising a more fragile Harper than post-fix Harper actually is. Sync Core #610 would land it. It cannot explain the 07-24 → 07-25 change (07-24 passed on the same fix-less core), so it is context, not cause.

Generated by Claude Opus 5.

🤖 Generated with Claude Code